Information processing apparatus and method, storage medium, and monitoring system

ABSTRACT

An apparatus for monitoring an object in a video received from at least one camera, comprises a storage unit storing a plurality of pieces of rule information each defining a condition of an object and a movement of the object to be observed in the condition, an input unit inputting information for identifying a monitored object and information representing a condition of the object, an acquiring unit acquiring rule information defining a movement to be observed for the object by referring to the storage unit based on the information representing the condition of the object, which is input by the input unit; and a monitoring unit determining whether the object in the video exhibits the movement to be observed, which is represented by the rule information acquired by the acquiring unit, and output a result of the determination.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an information processing apparatus andmethod, a storage medium, and a monitoring system.

Description of the Related Art

In recent years, a method of detecting a dangerous or abnormal conditionfrom a video is widely used for various application purposes. Forexample, as representative application purposes, a dangerous conditionof a patient or a resident is detected from a video of a camerainstalled in a hospital ward or a care facility, or an error in a workis detected from a video of a camera installed in a factory.

Of the conditions of a person, an object, a situation, and the like, acondition that is the detection target of a detecting system is called atarget condition in this specification. Detection based on a video isexecuted by a specific processing method corresponding to a targetcondition. For example, if “a condition in which an object is moving” isthe target condition, a processing method of, for example, performingprocessing such as object detection or motion vector estimation andperforming detection if the result exceeds a set threshold is executed.Such a definition of a condition such as a series of processing methodsor a parameter concerning detection based on a video will be referred toas a detection rule in this specification.

The detection rule is defined in correspondence with a target condition.However, an appropriate target condition sometime changes depending on aperson, an object, a situation, or the like. For example, in a facilitywhere only persons concerned can enter, an entry of an outsider is atarget condition, but an entry of a person concerned is not a targetcondition. In this case, the detection rule needs to be changed inaccordance with the target person or situation.

Japanese Patent No. 5845506 (to be referred to as a literature 1hereinafter) discloses a method of detecting a target condition set foreach person. In this method, a person and the situation of the personare estimated from a video. Next, the estimated situation is comparedwith a target condition set for the person, and if these match,detection is performed. This enables appropriate detection even in acase in which the target condition changes for each person.

International Publication No. 2007/138811 (to be referred to as aliterature 2 hereinafter) discloses a method of acquiring the actionauthority level of a person or a vehicle and detecting a targetcondition set for the action authority level. This method also enablesdetection in a case in which the target condition changes for eachperson or vehicle.

In the method of literature 1, a target condition set for each person isdetected from a video. However, there is neither a description nor asuggestion about setting a target condition. Hence, the method canhardly be applied in a case in which target condition setting itself isdifficult because, for example, a target condition corresponding to aperson is not self-evident or varies.

In the method of literature 2, a target condition set for each actionauthority level is detected from a video. However, as in literature 1,since the method does not set a target condition, there exists the sameproblem.

SUMMARY OF THE INVENTION

The present invention provides a technique capable of easily setting amovement of a target condition, which should be observed, even in a casein which it is difficult to set a condition of an object, and monitoringthe movement.

According to an aspect of the invention, there is provided aninformation processing apparatus for monitoring an object in a receivedvideo, comprising: a storage unit configured to store a plurality ofpieces of rule information each defining a condition of an object and amovement of the object to be observed in the condition; an input unitconfigured to input information for identifying a monitored object andinformation representing a condition of the monitored object; anacquiring unit configured to acquire rule information defining amovement to be observed for the monitored object by referring to thestorage unit based on the information representing the condition of themonitored object, which is input by the input unit; and a monitoringunit configured to determine whether the monitored object in thereceived video exhibits the movement to be observed, which isrepresented by the rule information acquired by the acquiring unit, andoutput a result of the determination.

According to the present invention, it is possible to easily set amovement of a target condition, which should be observed, even in a casein which it is difficult to set a condition of an object, and monitorthe movement.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the arrangement of a watching systemaccording to the first embodiment;

FIG. 2 is a view showing an example of the GUI of a condition rulesetting unit according to the first embodiment;

FIG. 3 is a view showing an example of the GUI of a data input unitaccording to the first embodiment;

FIG. 4 is a view showing an example of the GUI of a detection rulesetting unit according to the first embodiment;

FIG. 5 is a view showing an example of the GUI of a detection resultpresenting unit according to the first embodiment;

FIG. 6 is a flowchart showing an example of a detection rule settingprocedure according to the first embodiment;

FIG. 7 is a flowchart showing an example of a detection procedureaccording to the first embodiment;

FIG. 8 is a block diagram of the arrangement of a watching systemaccording to the second embodiment;

FIG. 9 is a view showing an example of the GUI of a detection settingcorrection unit according to the second embodiment;

FIG. 10 is a block diagram showing an example of the arrangement of awatching system according to the third embodiment;

FIG. 11 is a view showing an example of work instruction data accordingto the fourth embodiment;

FIG. 12 is a view showing an example of work instruction data accordingto the fourth embodiment;

FIG. 13 is a block diagram showing an example of the arrangement of awork management system according to the fourth embodiment; and

FIG. 14 is a view for explaining the outline of a movement according tothe fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Embodiments according to the present invention will now be described indetail with reference to the accompanying drawings. Note that in eachembodiment to be explained below, an example of application to awatching system will be described. The watching system according to thisembodiment defines a dangerous or abnormal condition of a patient in ahospital as a target condition and automatically sets a detection ruleof the target condition. The watching system further automaticallydetects the patient in the target condition from a video of a camerabased on the detection rule.

A hospital is required to be able to find, and quickly cope with, adangerous or abnormal condition of a patient, thereby preventingdeterioration of the condition or preventing it. However, it isdifficult to frequently confirm the condition of each patient because ofwork cost. In such a case, when a detecting system configured tocooperate with a camera and set a dangerous or abnormal condition of apatient as a target condition is introduced, the abnormal condition canbe expected to be found at low work cost.

However, the dangerous or abnormal condition may change for eachpatient. An example is walking. Walking is a normal condition for ageneral patient. However, walking for a patient who needs absolute restis an abnormal condition. To cope with such differences betweenpatients, the detection rule of the detecting system needs to be changeddepending on the patient.

On the other hand, however, when manually setting the detection rule foreach patient, the work cost for the setting of the detection rulebecomes high in proportion to the number of patients. In addition, sincethe condition of a patient changes in accordance with the progress of asymptom or a treatment situation, the detection rule also needs to bereset accordingly. It is therefore necessary to set the detection ruleat lower work cost.

As a characteristic feature, the watching system according to thisembodiment automatically sets a target condition and a detection rulecorresponding to a patient based on patient's medical data to bedescribed later, and detects the target condition based on theautomatically set detection rule.

Here, the medical data is, for example, data of a medical chartcorresponding to each patient and including personal information such asa name and a face image and at least one piece of medical informationsuch as a symptom or a care instruction. The medical data is not limitedto a specific form. For example, the medical data may be a document of aphysical medium such as paper or may be digitized.

Examples of medical information are the information of a disease name, asymptom, a morbid portion position, a surgery record, and a careinstruction. The form of the medical information is not limited to aspecific form. For example, the medical information may be an arbitraryor specific text or may be an attribute expressed as a category or avalue. Alternatively, the medical information may be a feature amount ordistribution expressed in a specific feature space.

As examples of a use method of the watching system according to thisembodiment, for example, a target condition of a hospital patient isdetected in a hospital ward, or system cooperation is establishedbetween a hospital and a patient's home, and a target conditioncorresponding to a diagnosis result in the hospital is detected in thehome.

First Embodiment

FIG. 1 is a block diagram showing an example of the arrangement of awatching system according to the first embodiment. As shown in FIG. 1,the system includes a detection rule storage device 1, a terminalapparatus 10, a setting apparatus 20, a video supplying device 30, and adetecting apparatus 40. Note that these apparatuses may be connected viaa network. As the network, for example, a fixed telephone line network,a portable telephone line network, the Internet, or the like isapplicable. In addition, the communication form can be either wired orwireless. These apparatuses may be included in one of the apparatuses.

The detection rule storage device 1 is a storage device configured tostore detection rules. As the detection rule storage device 1, a storagedevice such as a hard disk drive (HDD), a solid state drive (SSD), or anSD memory card can be applied. Note that a plurality of detection rulestorage devices 1 may exist and distributively store the detectionrules.

The detection rule according to this embodiment is a definitionincluding a processing method for detecting a target condition and theparameters of the processing. Here, the processing method defined by thedetection rule includes at least one processing module, the informationof the execution order and input/output of the processing modules, andthe conditions of detection determination.

Note that as a characteristic feature, the detection rule according tothis embodiment includes processing for detecting a target patient. Inthis embodiment, the definition of processing of detecting a targetpatient and parameters will particularly be referred to as a persondetection rule, and the definition of processing of detecting a targetcondition and parameters will be referred to as a condition detectionrule. In detection, first, a patient in a video is detected based on theperson detection rule, and a condition detection rule corresponding tothe detected patient is then executed.

The processing method of the person detection rule is not limited to aspecific method. For example, a method of collating face images may beused, or a method of reading an ID tag worn by a patient in combinationwith a sensor may be used. As an example in a case in which a face imageis collated, a processing method can be considered in which (1) a faceregion in a video is detected by face detection processing, (2) a localfeature amount is extracted from the image of the detected face region,and (3) detection is performed in a case in which the similarity to thefeature amount of a patient registered in advance is equal to or morethan a threshold. Additionally, in this case, the feature amount of thepatient, the threshold of similarity, and the like can be considered asparameters.

The processing method of the condition detection rule is not limited toa specific method, either. For example, when orientation estimation andmotion vector estimation exist as processing modules, a processingmethod can be considered in which (1) an orientation is estimated fromeach frame of a video, (2) the coordinates of limbs are extracted fromthe estimated orientation, (3) a motion vector is estimated for thecoordinates of the limbs based on preceding and subsequent frames, and(4) detection is performed in a case in which a motion equal to or morethan a threshold is estimated. Additionally, in this case, the size ofthe search range to estimate the motion vector, the threshold ofdetection determination, and the like can be considered as parameters.

In addition, an action and condition of a patient, for example, whethera patient is moving a specific part (for example, the right hand) orwalking using a cane may be estimated using a CNN (Convolutional NeuralNetwork).

Note that the detection rule may correspond to a single frame of a videoor may correspond to a plurality of frames. In addition, the detectionrule storage device 1 may store the personal information or the targetcondition of a patient corresponding to a detection rule in associationwith the detection rule.

The terminal apparatus 10 is a computer apparatus used by the user ofthe system, and includes an information presenting unit 11 and anoperation detecting unit 12. As the terminal apparatus 10, for example,an information processing apparatus represented by a personal computer(PC), such as a tablet PC, a smartphone, a feature phone, or a smartspeaker is applicable. Note that a plurality of terminal apparatuses 10may exist. In this case, the terminal apparatuses 10 may communicatewith each other and share information.

The information presenting unit 11 includes an output device such as animage display panel or a speaker, and presents information input fromthe setting apparatus 20. The information presenting means is notlimited to a specific method. For example, various kinds of userinterfaces (UIs) may be displayed on a screen display panel, orinformation may be converted into a voice and reproduced from a speaker.Note that the UI may be a command UI (CUI) or a graphical UI (GUI).

The operation detecting unit 12 includes an input device such as acontroller, a keyboard, a mouse, a touch panel, or a microphone, anddetects an operation of the user of the system and also outputsoperation information representing the detected operation to the settingapparatus 20. The input device of the operation detecting unit 12 andthe output device of the information presenting unit 11 may wholly orpartially be shared or connected or may be separated.

The setting apparatus 20 is an apparatus that sets the person detectionrule and the condition detection rule of a patient as a target based onmedical data input from an external apparatus, and outputs the rules tothe detection rule storage device 1. The medical data stored in theexternal apparatus includes not only a patient ID for identifying eachpatient and the medical chart (or medical information) of the patientbut also information used to identify the face of the patient. Note thatwhen an ID tag is used to identify a patient, a table representing thecorrespondence relationship between the ID tag and a patient ID isstored in an external storage device.

The setting apparatus 20 includes a condition rule storage unit 26, acondition rule setting unit 21, a data input unit 22, a contentanalyzing unit 23, a target condition setting unit 24, and a detectionrule setting unit 25. The setting apparatus 20 is formed by, forexample, an information processing apparatus such as a personalcomputer, and each processing unit may be implemented by a controlprogram or an application program executed by a CPU.

The condition rule storage unit 26 stores a condition rule (to bedescribed later) input from the condition rule setting unit 21, andoutputs it to the target condition setting unit 24. Here, the conditionrule is a definition including a processing method for outputting adangerous or abnormal condition as a target condition based on baseinformation to be described later and the parameters of the processing.

In this embodiment, the target condition is not limited to a specificform. For example, the target condition may be a text such as “movingthe right arm” or may be expressed by a combination of an item and oneor more values or ranges, for example, “target: right arm, movingamount: 10 or more”. In addition, the target condition may be expressedby a point, a set, a range, a distribution, a conditional expression, orthe like on a space having one or more coordinate axes corresponding toan item such as a moving amount or an arm angle. Furthermore, the targetcondition may be selected from a set of candidates of target conditions.

The base information according to this embodiment is information servingas a basis to decide the target condition. The base information is notlimited to a specific form. For example, the base information may be anarbitrary or specific text or may be a category or a value representingan attribute. In addition, the base information may be an image or avoice or a combination thereof.

Note that the watching system according to this embodiment uses medicalinformation as base information, and a case in which the baseinformation is medical information will be described below. However, thebase information in the embodiment is not limited to medicalinformation.

The form of the condition rule is not limited to a specific form. Forexample, as an example of the form of a condition rule, the danger levelof a condition is defined for each medical information. When medicalinformation and the threshold of the danger level are input, a conditioncorresponding to a danger level equal to or more than the threshold isoutput based on the medical information. In this case, it can beconsidered that, for example, if the medical information is “the rightleg is broken”, a danger level “80” for “walk without crutches”, adanger level “30” for “walk with crutches”, and the like are defined.When “50” is designated as the threshold of the danger level, “walkwithout crutches” is output as the target condition.

As another example of the form of the condition rule, it can beconsidered that when the information of a damaged part or the degree ofdamage of a person can be extracted from medical information, acondition in which a load on the damaged part exceeds an allowable rangeis output as the target condition. In this case, the condition ruledefines in advance the allowable range of a load based on the damagedpart and the degree of damage. As the processing, for example, pieces ofinformation representing that the damaged part is “right leg”, and thedegree of damage is “broken” are extracted from medical information suchas “the right leg is broken” based on corresponding keywords. Afterthat, a load on the damaged part is calculated for each condition, andif the load exceeds an allowable range defined in advance, the conditionis output as the target condition. Note that to calculate the load ineach condition, a load defined in advance may be referred to, or a loadmay be calculated by a physical simulation.

As still another example of the form of the condition rule, it can beconsidered that information concerning a restriction/prohibitionextracted from medical information is extracted, and a conditionviolating the restriction/prohibition is output as the target condition.In this case, in a case in which, for example, a keyword such as “mustnot” representing a prohibition is detected from medical information“the patient must not lie face down on the bed when sleeping”, acorresponding “condition in which the patient lies face down on the bedwhen sleeping” is set as the target condition.

As still another example of the form of the condition rule, it can beconsidered that in a case in which the target condition is expressed ona space having specific coordinate axes, the target condition is outputbased on an abnormal level generated based on the spatial distance to a“desirable condition” corresponding to medical information. In thiscase, the “desirable condition” corresponding to the medical informationmay be defined in advance or may be generated based on a keyword. Forexample, when the coordinate axes of the space representing a conditioncorrespond to the moving amounts of the parts of a human body, keywords“right arm” and “rest” are extracted from medical information “keep theright arm at rest”. As a corresponding “desirable condition”, a range inwhich the value of a coordinate axis corresponding to the moving amountof the right arm becomes equal to or less than a predetermined thresholdthat can be regarded as a rest is generated. When the desirablecondition is decided, a range in which the spatial distance becomesequal to or more than a predetermined threshold is output as the targetcondition. Note that the parameters of various kinds of thresholds suchhas the range of the magnitude of a motion that can be regarded as arest are also defined in advance by the condition rule.

Furthermore, as for the condition rule, the target condition may beoutput based on a plurality of pieces of medical information. Forexample, the threshold may be defined not as a constant but as adistribution corresponding to an age. Medical information concerning anage such as “80 years old” may be combined with medical informationconcerning an exercise restriction such as “vigorous exercises areprohibited”, and a condition corresponding to an exercise intensityregarded as a vigorous exercise in a case of 80 years old may be outputas the target condition.

In addition, a target condition such as “fall down the stairs” that doesnot depend on medical information may always be output.

The condition rule setting unit 21 sets (or creates) a condition ruleand outputs it to the condition rule storage unit 26. The condition rulestorage unit 26 can store a plurality of condition rules. A method ofsetting the condition rule is not limited to a specific method. Forexample, a GUI for the setting may be provided to cause the user of thesystem such as a medical worker to manually define the condition rule,or the condition rule may be defined using an expert system for medicaluse. Alternatively, the condition rule may be defined using a machinelearning method such as Deep Learning based on learning data formed by aset of medical information and a condition of a patient.

Note that the condition rule setting unit 21 sets the condition rulebefore input of medical data and does not perform individual setting foreach medical data. However, there is no limitation for setting thecondition rule after input of medical data.

FIG. 2 shows an example in a case in which the condition rule settingunit 21 provides a GUI. FIG. 2 shows a GUI configured to set a conditionrule to decide a target condition for each medical information based onthe danger level of each condition and a threshold. Reference numeralU21-1 denotes a list of registered medical information. Each row of thelist corresponds to medical information, and medical information to beset can be selected by clicking a check box at the left end. Referencenumeral U21-2 denotes an input field to register medical information.When an addition button on the right side is clicked in a state in whichthe text of medical information is input to the text field, the medicalinformation can additionally be registered. Reference numeral U21-3denotes a list of registered conditions and danger levels. Each row ofthe list corresponds to each condition, and the danger level of eachcondition can be input as a numerical value in a danger level inputfield at the right end. Note that the danger level is independently setfor each medical information selected by reference numeral U21-1.Reference numeral U21-4 denotes an input field to register a condition.Here, a condition can additionally be registered, like reference numeralU21-2. Reference numeral U21-5 is a field to input the threshold of adanger level. When a numerical value is input here, the threshold of adanger level as a target condition is set.

A detailed example of setting a condition rule by the user of the systemusing the GUI shown in FIG. 2 will be described. Here, the conditionrule outputs the target condition based on medical information and thethreshold of a danger level. First, the user of the systeminputs/registers medical information such as “the right leg is broken”or “the left leg is broken” using the text form. Next, a condition suchas “walk without crutches”, “hit the right leg”, or “hit the left leg”,is registered like the medical information. After that, the danger levelof each condition is input as a value of “0” or more for each medicalinformation. Note that as the danger level of each condition, “0” may beinput as an initial value. In addition, the threshold of the dangerlevel of the detection target is input. The condition rule is thus set,which outputs, as a target condition, a condition equal to or more thanthe threshold of a danger level for each medical information.

However, the GUI of the condition rule setting unit 21 is not limited tothe example shown in FIG. 2, and an arbitrary GUI may be used. Forexample, an image representing a human body may be displayed, and anicon representing a symptom is superimposed on a portion correspondingto a morbid portion, thereby expressing medical information. Inaddition, in a case in which the target condition is expressed on aspace, a slider component corresponding to each coordinate axis of thespace may be provided, and the target condition may be defined byoperating the slider components.

The data input unit 22 acquires, from an external apparatus, medicaldata that is the input to the setting apparatus 20, and outputs themedical data to the content analyzing unit 23. The medical data may beacquired from a database such as an electronic medical chart system or afile of a predetermined format such as CSV, or may be acquired byproviding a GUI to input medical data. When the GUI is provided, theuser of the system may input and confirm the contents using the terminalapparatus 10. Alternatively, instead of receiving medical data from theexternal apparatus, a physical medium such as paper with medical datamay be converted into an image using an input device such as a scanneror a camera, and the medical data may be acquired using a recognitionmethod such as OCR (Optical Character Recognition/Reader). Note that themedical data acquired by the data input unit 22 may be stored by anarbitrary storage unit.

FIG. 3 shows an example in a case in which the data input unit 22provides an electronic medical chart input GUI. FIG. 3 shows anelectronic medical chart GUI capable of inputting the personalinformation of a patient and a care instruction. Reference numeral U1-1denotes a text field to input the name of a patient. Reference numeralsU1-2 and U1-3 denote pulldown menus to input the sex and age of thepatient respectively. Reference numeral U1-4 denotes a text area toinput the text of an instruction to a nurse. Reference numeral U1-5denotes an OK button representing completion of input of the electronicmedical chart GUI and a cancel button representing input discard.Reference numeral U1-6 denotes a patient ID used to specify one patient(because there is the possibility of the same name).

However, the GUI of the data input unit 22 is not limited to the exampleshown in FIG. 3, and an arbitrary GUI can be used. For example, the GUImay include components to input the face image, symptom, and treatmentrecord of the patient. In addition, connection to an external medicalinformation system may be selected on the GUI. A different UI other thanthe GUI may be used.

Note that the data input unit 22 may include a GUI configured to set afixed form pattern of partial or whole medical data and enable input ofmedical data by selecting one or a plurality of fixed form patterns.

The content analyzing unit 23 analyzes medical data input from the datainput unit 22 and outputs extracted medical information to the targetcondition setting unit 24 and the personal information of the patient tothe detection rule setting unit 25. As the method of analyzing themedical data, a method using a recognition method such as textrecognition such as LSA (Latent Semantic Analysis) or image recognitionmay be used, or a method of searching for a word corresponding to apredetermined keyword and extracting it may be used.

In addition, an expression such as an abbreviation or an orthographicalvariant in medical data may be normalized to a unified expression inadvance, or a symbol may be converted into a language. For example, whenmedical information includes a text “Immediately after surgery on theright arm, keep the right arm at rest and take A medicine after meals”,examples of medical information to be extracted are “immediately aftersurgery on the right arm”, “keep the right arm at rest”, and “take Amedicine after meals”.

A detailed example of processing performed by the content analyzing unit23 will be described. Here, assume that medical information candidatesare defined in advance in the form of texts such as “the right leg isbroken” and “cannot roll over by himself/herself” on the GUI shown inFIG. 2 or the like. At this time, the content analyzing unit 23 appliesthe LSA to each medical information and extracts the latent meaning ofeach medical information. Next, the medical data is disassembled intosentences such as “fell down the stairs and broke the right leg”, theLSA is applied, and the latent meaning of each sentence of the medicaldata is extracted. After that, the similarity of latent meanings betweenthe medical information and the medical data is calculated, and medicalinformation having a high similarity to the latent meaning of themedical data is extracted as medical information corresponding to themedical data. For example, when “fell down the stairs and broke theright leg” is given as medical data, medical information such as “theright leg is broken” with a high similarity is extracted.

Upon receiving the medical information from the content analyzing unit23, the target condition setting unit 24 automatically searches thecondition rule storage unit 26 for a condition rule matching the medicalinformation, and outputs the matching condition rule to the detectionrule setting unit 25. As the method of setting a target condition, themedical information is input to a processing method defined in eachcondition rule, and an output condition is set as the target condition.If there exist a plurality of pieces of medical information, eachcondition output in correspondence with each medical information may beset as a target condition. Note that if no condition is output, a targetcondition need not be set.

For example, in a case in which the condition rule defines that whenmedical information and the threshold of a danger level are input, acondition in which the danger level becomes equal to or more than thethreshold under the medical information is output, the target conditionsetting unit 24 inputs medical information and the threshold of thedanger level to the condition rule. The threshold of the danger levelmay be set by providing a GUI that allows the user of the system to doinput, or a predetermined value defined in advance may be used. If aplurality of pieces of medical information exist in correspondence withone patient, the condition rule is applied to each medical informationto set a plurality of target conditions.

Note that the target condition setting unit 24 may provide a GUI thatdisplays a set target condition and allows the user to select whether touse it as a target condition, and output only a target conditionapproved by the user of the system. Furthermore, in a case in which aplurality of condition rules exist, a GUI capable of selecting acondition rule to be applied may be provided, and the user of the systemmay arbitrarily select a condition rule. For example, there may exist aplurality of condition rule sets that are the sets of condition rulessuch as condition rules for terminal care and condition rules for asurgical ward. A GUI configured to display the outline of each conditionrule set and allow the user of the system to select a condition rule setmay be provided. In addition, a condition rule to be applied may beselected from a plurality of condition rules based on the medicalinformation.

The detection rule setting unit 25 sets a detection rule based on thepersonal information of the patient input from the content analyzingunit 23 and the target condition input from the target condition settingunit 24, and outputs the detection rule to the detection rule storagedevice 1. As a characteristic feature, the detection rule according tothis embodiment includes a person detection rule and a conditiondetection rule. The detection rule setting unit 25 sets a persondetection rule corresponding to the personal information and a conditiondetection rule corresponding to the target condition.

The person detection rule according to this embodiment is set based onthe personal information of the patient. However, the processing methodof the person detection rule is not limited to a specific form. Forexample, the face image of the patient prepared in advance may becompared/collated with a face region in a video, or an ID tag worn bythe patient may be detected by cooperation with a sensor. In addition,the processing method of the person detection rule may be setindividually for each patient, or may be commonly set.

For example, in a case in which the processing method of the persondetection rule uses collation of a face image, and is set commonly forall patients, the detection rule setting unit 25 extracts a face imagefrom the personal information of the patient and sets a parameterrepresenting a feature of the face to be used for collation, therebysetting the person detection rule. Note that as the parameterrepresenting the feature of the face, for example, a SIFT feature amountor the like can be used. In this case, the detection processing isperformed based on the similarity between a feature amount extractedfrom a video and a feature amount extracted in advance.

The condition detection rule setting method according to this embodimentis not limited to a specific method. For example, every time a targetcondition is input, an appropriate condition detection rule may beestimated. Alternatively, a condition detection rule corresponding toeach target condition may be set in advance, and a condition detectionrule corresponding to an input target condition may be referred to. Inaddition, the condition detection rule may automatically be set by thesystem, or may manually be set by the user of the system. Furthermore,these may be combined.

As an example of the condition detection rule setting method, in a casein which, for example, the target condition is expressed by acombination of one or more attributes such as “part: right arm, movingamount: 10 or more”, a method of setting processing corresponding toeach attribute and thus setting a condition detection rule is usable. Inthe case of this example, the target condition is defined by theattributes of a human body part and a moving amount. For example, it ispossible to automatically set orientation estimation processing asprocessing corresponding to the part attribute and motion vectorestimation processing as processing corresponding to the moving amountattribute. When the condition detection rule is thus set, detectionprocessing of performing detection when the moving amount of the rightarm portion is 10 or more can be executed.

Additionally, for example, in a case in which the target condition isexpressed by a specific feature amount, processing of extracting acorresponding feature amount and processing of obtaining a similarity bycomparing the extracted feature amount with the target condition areset, thereby automatically setting the condition detection rule.

As an example of the method of manually setting the condition detectionrule, a method of setting a condition detection rule corresponding toeach target condition by a GUI can be used. FIG. 4 shows an example of aGUI provide by the detection rule setting unit 25 in this case. FIG. 4shows a GUI that causes the user of the system to set a conditiondetection rule corresponding to each target condition. Referencenumerals U25-1 and U25-2 denote tables showing target conditions, andeach row corresponds to a target condition. Reference numeral U25-1denotes a column of checkboxes configured to select a target to set acondition detection rule. In reference numeral U25-1, one or a pluralityof target conditions can be selected. In a case of selecting a pluralityof target conditions, a common condition detection rule is set for theselected target conditions. Reference numeral U25-2 denotes a column todisplay texts representing the target conditions. Reference numeralsU25-3 to U25-6 denote pulldown menus and text fields used to set acondition detection rule. Reference numerals U25-3 to U25-6 correspondto the processes and parameters of a condition detection rule, and theyare executed in order. However, if the processes are independent, theymay be executed in random order or in parallel. Note that the parameterof each processing changes in accordance with the selected processing.The parameter U25-3 represents coordinate estimation processing of ahuman body part. As the parameter of the target part, the right arm isset. The parameter U25-4 represents motion vector estimation processing.Here, as the parameters of target coordinates, the processing result ofthe parameter U25-3, that is, the estimated coordinates of the right armare set. The parameter U25-5 represents processing of conditionbranching in a case in which an input value is equal to or more than athreshold. The output of the parameter U25-4, that is, the motion vectorof the right arm portion is set as the parameter of the input value, and“20” is set as the parameter of the threshold. The parameter U25-6 isexecuted only in a case in which the conditions defined by the parameterU25-5 are satisfied, and represents that detection processing isperformed. These processes can be changed by selecting the processes orthe parameters from the pulldown menus and text fields. However, the GUIused to set the condition detection rule is not limited to the exampleshown in FIG. 4, and an arbitrary GUI may be used. For example, acomponent configured to visualize a target condition by an image or avideo and display it may be included. In addition, a button componentconfigured to add or delete a processing step may be included.

A detailed example of setting the condition detection rule of a targetcondition using the GUI shown in FIG. 4 will be described. Here, assumethat the condition detection rule is independently set for each targetcondition, and an arbitrary number of processing steps are executed inorder. First, the user of the system selects a target condition to set acondition detection rule on the GUI. Next, each process to be executedby the condition detection rule is set for each processing step.Candidates of processes executable in each processing step are definedon the system in advance. The user of the system selects a processcandidate and inputs a parameter. For example, for a target condition“walk without crutches”, a condition detection rule shown below is set.This condition detection rule defines processes and parameters to detectthe target condition when walking is detected by action recognition, andcrutches are not detected by object detection.

(1) Process [action recognition], parameter [(target action) walk]

(2) Process [object detection], parameter [(object) crutches]

(3) Process [condition branching], parameter [(condition) process 1 fordetection success, and process 2 for detection failure]

(3-1) Process [detection notification]

Here, the process of condition branching indicates processing ofexecuting the next processing step only in a case in which the conditionis satisfied, and interrupting the processing in a case in which thecondition is not satisfied. In addition, the process of detectionnotification is processing of outputting, to the system, informationrepresenting that the target condition is detected. Note that theprocesses have different parameters. When the user of the system selectsa process, an input field of the corresponding parameter is displayed onthe GUI.

Note that in a case in which the condition detection rule correspondingto the target condition cannot be set by the detection rule setting unit25, a message representing that the setting of the detection rule hasfailed may be displayed on the information presenting unit 11 of theterminal apparatus 10 or the like, thereby notifying the user of thesystem of it.

The video supplying device 30 is a device configured to acquire a videoand output the acquired video to the detecting apparatus 40. The videosupplying device 30 and the detecting apparatus 40 are connected via anetwork for communication. The communication form can be either wired orwireless. The video supplying device 30 may include an image capturingdevice such as a network camera, a web camera, or a wearable camera andacquire and output a video of the environment in which the device isinstalled, or may include a storage device such as an HDD or an SSD andoutput a stored video. In addition, when acquiring a video, the videosupplying device 30 may acquire the information of the image capturingenvironment such as the place where the video is captured and the angleof field and output the information in association with the video. Inaddition, a plurality of video supplying devices 30 may exist, and aplurality of videos may be output to the detecting apparatus 40.

The detecting apparatus 40 is an apparatus configured to detect thetarget condition corresponding to the patient based on the detectionrule and notify the detection result. The detecting apparatus 40includes a video acquiring unit 41, a detection determining unit 42, anda detection result presenting unit 43 shown in FIG. 1.

The video acquiring unit 41 acquires a video from the video supplyingdevice 30, and outputs it to the detection determining unit 42. Thedetection determining unit 42 performs detection processing based on thevideo input from the video acquiring unit 41 and a detection rulereferred to from the detection rule storage device 1, and outputs adetection result to the detection result presenting unit 43.

The detection determining unit 42 first executes the processing of theperson detection rule and detects that a registered patient is includedin the video. If the registered patient is detected, processing of thecondition detection rule corresponding to the patient is executed. Afterthat, a target condition detected as the detection result is output tothe detection result presenting unit 43. However, if a registeredpatient or target condition is not detected, information representingthat nothing is detected is output as the detection result.

The detection result presenting unit 43 provides a GUI configured tovisualize the detection result input from the detection determining unit42 and displays it on the information presenting unit 11 of the terminalapparatus 10, thereby presenting the detection result to the user of thesystem. As for the contents to be displayed, not only thepresence/absence of detection but also a plurality of pieces ofassociated information such as information concerning the targetcondition and the instruction of an action that the user of the systemshould make may be presented. In addition, the detection result may beconverted into a voice and presented by a speaker or the like. Thedetection result presenting unit 43 may further include a GUI configuredto do settings of the detection result presenting method and contentssuch that the user of the system can set the presenting method andcontents.

FIG. 5 shows an example of the GUI provided by the detection resultpresenting unit 43. FIG. 5 shows a GUI configured to superimpose theinformation of a detection result on a video at the time of detection.Reference numeral U43-1 denotes a window that displays the video orimage at the time of detection. When the window U43-1 displays thecaptured video, buttons such as play and pause buttons used to controlthe video may further be provided and displayed. Reference numeral U43-2denotes a frame indicating a portion in the video or image correspondingto the detection result. When the window U43-1 displays the video, theframe U43-2 also deforms and moves along with a change of the video.Reference numeral U43-3 denotes a popup component that displays detailsof the detection result. Examples of contents displayed by the popupcomponent U43-3 are a position in medical data corresponding to thedetection result, the target condition, and the detection rule.

Note that although FIG. 1 shows the setting apparatus 20, the detectionrule storage device 1, and the detecting apparatus 40 as separateapparatuses, these may be implemented by one apparatus. In addition, theapparatus may include the terminal apparatus 10.

FIG. 6 is a flowchart showing an example of a detection rule settingprocedure according to the first embodiment. The detection rule settingprocedure according to the first embodiment will be described below withreference to FIG. 6.

1. The condition rule setting unit 21 sets a condition rule. Thisprocessing corresponds to step S101 in FIG. 6.

2. The data input unit 22 inputs medical data of a patient. Thisprocessing corresponds to step S102 in FIG. 6.

3. The content analyzing unit 23 analyzes the contents of the medicaldata and extracts personal information and medical information. Thisprocessing corresponds to step S103 in FIG. 6.

4. The target condition setting unit 24 sets a target condition based onthe condition rule and the medical information. This processingcorresponds to step S104 in FIG. 6.

5. If the detection rule storage device 1 does not store a persondetection rule corresponding to the personal information, the detectionrule setting unit 25 sets a person detection rule. This processingcorresponds to steps S105 and S106 in FIG. 6.

6. If the detection rule storage device 1 does not store a conditiondetection rule corresponding to the target condition, the detection rulesetting unit 25 newly sets a condition detection rule. This processingcorresponds to steps S107 and S108 in FIG. 6.

7. The detection rule storage device 1 stores the person detection ruleand the condition detection rule corresponding to the patient. Thisprocessing corresponds to step S109 in FIG. 6.

With the above-described processing, a detection rule corresponding tothe medical data of the patient can be set. However, the processingprocedure according to this embodiment is not limited to that describedabove. For example, candidates of condition detection rules may be setin advance before the medical data is input.

FIG. 7 is a flowchart showing an example of a detection procedureaccording to this embodiment. The processing procedure will be describedbelow with reference to FIG. 7.

1. The video acquiring unit 41 acquires a video from the video supplyingdevice 30. This processing corresponds to step S201 in FIG. 7.

2. The detection determining unit 42 acquires a detection rule from thedetection rule storage device 1. This processing corresponds to stepS202 in FIG. 7.

3. The detection determining unit 42 detects a patient from the videobased on the person detection rule and identifies the patient. Thisprocessing corresponds to step S203 in FIG. 7.

4. If no patient is detected, the processing is ended. This processingcorresponds to step S204 in FIG. 7.

5. The detection determining unit 42 detects the target condition of thepatient in the video based on a condition detection rule correspondingto the patient. This processing corresponds to step S205 in FIG. 7.

6. The detection result presenting unit 43 presents the detection resultto the user of the system. This processing corresponds to step S206 inFIG. 7.

As described above, according to the first embodiment, it is possible toset a detection rule based on medical data and detect a target conditionfrom a video. However, the detection rule setting method and thedetection method described here are merely examples and are not limitedto these examples.

For easier understanding of the movement according to the firstembodiment, a description will be made again with reference to FIG. 14.Note that although FIG. 14 corresponds to FIG. 1, several components arenot illustrated for the sake of simplicity.

The condition rule storage unit 26 stores a plurality of pieces of ruleinformation each defining a condition (for example, a disease name suchas right leg fracture) of an object and a movement (for example, walkwithout crutches) that derives from the condition and should be observedconcerning the object (FIG. 2).

Assume that the content analyzing unit 23 newly receives information(medical data of patient ID: P00123) representing the condition of amonitored object. The content analyzing unit 23 analyzes the medicaldata and detects a result (a patient ID and a disease name). The targetcondition setting unit 24 receives the analysis result, and acquiresrule information matching the condition obtained by the analysis from anumber of condition rules registered in the condition rule storage unit26. Based on the information from the target condition setting unit 24,the detection rule setting unit 25 registers, in a detection ruledatabase 1 a of the detection rule storage device 1, information thatdefines a movement to be observed for the monitored object (here,patient ID: P00123).

Assume that the video acquiring unit 41 receives a video of an object (aperson walking in the hospital) from a certain video supplying device30. Note that at this stage, whether the object (person) in the video isa patient does not matter. The detection determining unit 42 extractsfeature information from the object and compares it with featureinformation registered in a monitored object database (patient database)1 b in the detection rule storage device 1, thereby determining whetherthe object in the video is a monitored object (one of registeredpatients). If the object in the video is a non-monitored object,processing is not performed anymore. If it is found that the object inthe video is a monitored object, the detection determining unit 42determines that the object in the video to be input from then on is themonitored object, and starts monitoring (tracking) of its movement.During monitoring processing, the detection determining unit 42determines whether the condition (the patient of the patient ID: P00123walks without crutches, or the right leg hits an object) to be observedfor the monitored object in the detection rule database 1 a is obtained,and supplies the determination result to the detection result presentingunit 43. Based on this information, in a case in which the monitoredobject performs the movement to be observed (in a case in which one ofwalking of the patient of the patient ID: P00123 without crutches andhit of the right leg against an object is detected), the detectionresult presenting unit 43 determines this as abnormality, and presentsthe detection result to the user of the system.

As a result, in a case in which the information representing thecondition of the monitored object is received, it is possible toautomatically decide rule information representing a movement to beobserved in correspondence with the condition without performing aspecific operation or setting of the object. In addition, it is possibleto monitor, for each monitoring target, whether there is a movement tobe observed, which is unique to the monitoring target.

Second Embodiment

In the first embodiment, an in-hospital watching system configured toset a detection rule corresponding to a patient has been described. Inthe second embodiment, a case in which processing is added to thein-hospital watching system according to the first embodiment toautomatically update the detection rule in accordance with updating ofmedical data will be described. A case in which the user of the systemcorrects an automatically set target condition or detection rule willalso be described.

In the hospital, the target condition to be detected sometimes changesdue to a change in the treatment situation or symptom of a patient. Forexample, for a patient who needs absolute rest, a condition in which thepatient walks about in the hospital is the target condition. However,the detection becomes unnecessary when the patient recovers and ispermitted to walk about in the hospital. At this time, if the detectionrule is automatically updated in accordance with updating of the medicaldata, the detection rule need not be reset explicitly. It is thereforepossible to reduce the work man-hour.

The watching system according to the second embodiment includes anarrangement common to the watching system according to the firstembodiment. Hence, in the second embodiment, portions different from thefirst embodiment will be described.

FIG. 8 is a block diagram showing an example of the arrangement of thewatching system according to the second embodiment. As shown in FIG. 8,the watching system according to the second embodiment includes adetection rule storage device 1, a terminal apparatus 10, a settingapparatus 20 a, a video supplying device 30, and a detecting apparatus40. Note that these apparatuses may be connected via a network. As thenetwork, for example, a fixed telephone line network, a portabletelephone line network, the Internet, or the like is applicable. Inaddition, these apparatuses may be included in one of the apparatuses.

The setting apparatus 20 a is an apparatus that sets a target conditionand a detection rule based on medical data input from the outside, andoutputs them to the detection rule storage device 1, like the settingapparatus 20 according to the first embodiment. As a characteristicfeature, the setting apparatus 20 a includes a condition rule storageunit 26, a condition rule setting unit 21, a data input unit 22, acontent analyzing unit 23, a target condition setting unit 24, and adetection rule setting unit 25, like the setting apparatus 20 accordingto the first embodiment, and also includes a detection rule updatingunit 27 and a detection setting correction unit 28, as shown in FIG. 8.

The detection setting correction unit 28 provides a GUI configured tocause the user of the system to confirm and correct contents set by thecomponents of the setting apparatus 20 a, and replaces a detection rulestored in the detection rule storage device 1 with a corrected detectionrule. Targets to be corrected by the setting apparatus 20 a are medicalinformation extracted by the content analyzing unit 23, a targetcondition set by the target condition setting unit 24, and a detectionrule set by the detection rule setting unit 25. The detection settingcorrection unit 28 may provide a GUI capable of correcting all of them,or may provide a GUI configured to correct some of them.

As an example of the GUI provided by the detection setting correctionunit 28, for example, a GUI as shown in FIG. 9 can be used. FIG. 9 showsa GUI that displays a target condition and a detection rule and allowsthe user of the system to correct them. Reference numerals U26-1 toU26-4 denote items capable of displaying and selecting targetconditions, and each row corresponds to a target condition. The itemU26-1 shows medical information used to set a detection rule. The itemU26-2 is a column of checkboxes to select a target condition. The itemU26-3 is a region to display a target condition. A period for eachtarget condition is also displayed in the region U26-3, and a targetcondition in a corresponding period is a target of detection. Note thatthe period of each target condition is set based on the detection ruleupdating unit 27 to be described later. When the period of a targetcondition is selected by double-click or the like, an editing conditionis set, and the period can be corrected. The button U26-4 is prepared todelete each target condition. When the button U26-4 is clicked, thetarget condition and the corresponding detection rule are deleted.Reference numeral U26-5 denotes an image or a video representing anexample of a target to be detected by a detection rule. For example,this is an image representing a human body or an icon showing acondition such as a movement. This icon may be superimposed to show thecondition of a corresponding part at the position of the icon. Inaddition, an animation showing a condition of detection may bereproduced. Reference numerals U26-6 and 26-7 denote a slider and a textfield used to adjust the parameters of the detection rule correspondingto the selected target condition. The type of the parameter settable herchanges depending on the type of the parameter included in the detectionrule. In addition, when the parameter of the detection rule iscorrected, the detected condition represented by the slider U26-6 alsochanges in synchronism. The slider U26-6 is a slider configured to setthe threshold of the magnitude of the movement to be detected. Thefrequency of detection is set in the text field U26-7. By the textfield, the interval of time to execute detection processing can be set.

Note that in a case in which a detection setting is changed by the userof the system, the detection setting correction unit 28 may correct thebehavior of each component of the setting apparatus 20 a. For example,if a target condition “the right arm is moving” corresponding to medicalinformation “keep the right arm at rest” is deleted, the correspondingcondition rule may be corrected and thus changed so the same targetcondition is not set after that. In addition, when the parameter of adetection rule is corrected, changing may be performed so as to set thedetection rule of the corrected parameter for the same target conditionafter that.

The detection rule updating unit 27 updates (resets) the detection rulebased on the medical data input from the data input unit 22, the medicalinformation input from the content analyzing unit 23, and the like. Thetiming (to be referred to as an updating timing hereinafter) of updatingthe detection rule is decided by some or all of methods to be describedbelow.

As one of the methods of deciding the updating timing, there is a methodof performing updating when updating of medical data is detected. Inthis case, since the detection rule is set based on the latest medicaldata, it is possible to quickly cope with a change in medicalinformation.

The method of detecting updating of medical data is not limited to aspecific method. For example, the updating date/time of medical data maybe stored, and updating may be detected in a case in which the latestupdating date/time is newer than the stored date/time. Alternatively,medical data may be input to the content analyzing unit 23, and updatingmay be detected in a case in which output medical information changes.

As another method of deciding the updating timing, there exists a methodof extracting the information of a period such as the date/time of startor the date/time of end from medical information and making the updatingtiming match the start and end of the period. For example, in a case inwhich medical information “absolute rest for two days” is input, thedetection rule is updated using medical information other than “absoluterest” two days after the date/time of input of the medical information.

Additionally, even in a case in which an explicit period is notdesignated, the period may be estimated based on the medical informationconcerning the symptom or treatment method. The period corresponding tothe symptom or medical method may be set manually in advance, or may beestimated by referring to a database concerning a general treatmentperiod of the symptom. For example, in a case in which there is medicalinformation “infected by influenza”, and a setting “the treatment periodof influenza is about one week” is manually done, the detection rule isupdated based on medical information other than “infected by influenza”at a timing after the elapse of one week.

When the detection rule updating unit 27 detects updating of medicaldata, the updated medical information is output to the content analyzingunit 23. If the medical information is changed, the content analyzingunit 23 outputs the changed medical information to the target conditionsetting unit 24. The target condition setting unit 24 sets a targetcondition based on the changed medical information. In a case in whichthe target condition changes as compared to that before the medicalinformation changes, or the target condition changes by a predeterminedamount or more, the target condition is output to the detection rulesetting unit 25. The detection rule setting unit 25 resets the detectionrule and outputs it to the detection rule storage device 1. Thedetection rule storage device 1 replaces the past detection rule withthe reset detection rule and stores it.

If the detection rule is reset along with the updating of the medicalinformation, the detection rule updating unit 27 may present thedetection rule changed by the resetting to the user of the system by aGUI or the like. In addition, the user of the system may be confirmedabout whether to approve the resetting of the detection rule, and thedetection rule may be replaced only when the updating is approved.

In the above-described way, it is possible to detect updating of medicaldata and reset the detection rule. In addition, the user of the systemcan confirm and reset the detection rule. However, the example describedin this embodiment is merely an example and is not limited.

Third Embodiment

In the third embodiment, a case in which processing is added to thewatching system according to the first embodiment to adjust a detectionrule in accordance with the image capturing environment of a video willbe described.

In a detecting system that handles videos of a plurality of cameras, theimage capturing conditions of the videos may be different. Here, theimage capturing conditions are various kinds of conditions including theposition and orientation of a camera, parameters of a camera such as aframe rate, and environmental conditions such as illumination. At thistime, even in a case in which the same target condition should bedetected, the detection rule is sometimes preferably changed for eachvideo.

For example, assume that a detection rule that defines performing motionvector estimation and performing detection based on the magnitude of amoving amount is set. In this case, if the frame rate of a videochanges, the amount of the estimated motion vector also changes. Hence,it is difficult to perform appropriate detection.

As a characteristic feature, the watching system according to the thirdembodiment changes a detection rule in correspondence with the imagecapturing conditions of a video, thereby performing detection processingsuitable for each video.

The watching system according to the third embodiment includes anarrangement common to the watching system according to the firstembodiment. Hence, in the third embodiment, portions different from thefirst embodiment will be described. FIG. 10 is a block diagram showingan example of the arrangement of the watching system according to thethird embodiment. As shown in FIG. 10, the work management systemaccording to the third embodiment includes a detection rule storagedevice 1, a terminal apparatus 10, a setting apparatus 20, a videosupplying device 30 b, and a detecting apparatus 40 b. Note that theseapparatuses may be connected via a network. As the network, for example,a fixed telephone line network, a portable telephone line network, theInternet, or the like is applicable. In addition, these apparatuses maybe included in one of the apparatuses.

The video supplying device 30 b is a device configured to acquire avideo and output the acquired video to the detecting apparatus 40 b,like the video supplying device 30 according to the first embodiment.The video supplying device 30 b includes an image capturing conditionacquiring unit 31 in addition to the function of the video supplyingdevice 30 according to the first embodiment. The image capturingcondition acquiring unit 31 acquires an image capturing condition foreach video supplied by the video supplying device 30 b and outputs theimage capturing condition to a video acquiring unit 41 b of thedetecting apparatus 40 b.

The method of acquiring the image capturing condition is not limited toa specific method. For example, in a case in which the video supplyingdevice 30 b includes a camera device and acquires a video from thecamera device, a method of accessing the camera device to acquire imagecapturing parameters such as a frame rate and a resolution can be used.In addition, in a case in which the video supplying device 30 b acquiresvideo file data, a method of acquiring an attribute concerning the imagecapturing condition from the information of the attribute included inthe video file data can be used. In addition, the image capturingcondition acquiring unit 31 may provide a GUI used to input an imagecapturing condition for each video and cause the user of the system todirectly input the image capturing condition to acquire the imagecapturing condition.

The detecting apparatus 40 b is an apparatus configured to detect atarget condition based on a detection rule and notify the detectionresult, like the detecting apparatus 40 according to the firstembodiment. The detecting apparatus 40 b includes a detection resultpresenting unit 43 shown in FIG. 10 and also includes a video acquiringunit 41 b and a detection determining unit 42 b.

The video acquiring unit 41 b acquires a video and an image capturingcondition from the video supplying device 30, and outputs them to thedetection determining unit 42 b, like the video acquiring unit 41according to the first embodiment.

The detection determining unit 42 b performs detection processing basedon the video and the image capturing condition input from the videoacquiring unit 41 b and the detection rule referred to from thedetection rule storage device 1, and outputs the detection result to thedetection result presenting unit 43. As a characteristic feature, thedetection determining unit 42 b changes the detection rule incorrespondence with the image capturing condition. However, the methodof changing the detection rule is not limited to a specific method.

As an example of the method of changing the detection rule by thedetection determining unit 42 b, a method of changing input/output of aspecific process in the detection rule in correspondence with the imagecapturing condition can be used. For example, in a case in which “motionvector estimation processing” is executed by the detection rule, amethod of dividing the motion vector amount to be output by the framerate is used. As the characteristic of motion vector estimationprocessing, the estimation amount of the motion vector is reverselyproportional to the frame rate. Hence, the motion vector amount can benormalized with respect to the frame rate by dividing it by the framerate. In addition, for example, processing of converting the resolutionof a video into a predetermined size may be added.

In the above-described way, even if the image capturing conditionchanges between videos, appropriate detection processing can be executedby automatically changing the detection rule. That is, it is possible toperform detection processing based on one detection rule for videos froman unspecified number of video supplying devices with different imagecapturing conditions.

Fourth Embodiment

The fourth embodiment will be described. In the fourth embodiment, acase in which a dangerous or abnormal condition of a worker or a machineis detected from a video of a camera based on a detection ruleautomatically set in correspondence with a work in a factory will bedescribed as an example.

In a factory, there is a demand to be able to find a dangerous orabnormal condition of a worker or a machine and quickly cope with it,thereby preventing an occurrence of an accident or a defective product.However, it is difficult to frequently confirm the state of a workbecause of work cost. In this case, a detecting system that cooperateswith a camera is introduced, and a dangerous or abnormal condition of aworker or a machine is set as a target condition, so that the conditioncan be expected to be automatically detected.

A dangerous or abnormal condition in the factory can be detected by thesame arrangement and processing as the watching system described in thefirst embodiment. However, the work management system according to thisembodiment is different from the watching system according to the firstembodiment in the following points.

First, to set a detection rule, not medical data but a work instructiondata to be described later is used. Next, the detection rule is set notfor each patient but for each work performed by a worker. Furthermore, acondition rule can automatically be added based on the work instructiondata to be described later.

The work instruction data according to the fourth embodiment is data ofan incident report or a work standard document including workinstruction information that is the information of an instructionconcerning a work method. Examples of the work instruction informationis a work category such as “roller assembling”, a work order, a note foreach work such as “measure an assembling position by a scale”, and theinformation of a qualification needed for each work. Note that the workinstruction data or work instruction information is not limited to aspecific form and can be an arbitrary form, like the medical data ormedical information according to the first embodiment.

FIG. 11 shows an example of work instruction data. FIG. 11 shows anexample of a work standard document to attach a sack to a tube andcharge a powder. A field WS-1 stores information representing the orderof works. A field WS-2 stores information representing detailed workcontents such as “place a box under a tube” and “inject a powder”. Afield WS-3 stores the information of remarks such as a note for eachwork, for example, “remove static electricity in advance”. The fieldWS-3 may have no information described depending on the work.

FIG. 12 shows another example of work instruction data. FIG. 12 shows anexample of a report of an incident that has occurred during a powdercharging work. Reference numeral HR-1 denotes a date/time of theoccurrence of an incident; HR-2, a work performed at the time of theincident; and HR-3, contents of the incident. Reference numeral HR-4denotes a cause of the incident; HR-5, a measure to prevent theoccurrence of the incident; and HR-6, a photo of the site where theincident has occurred in which an arrow indicates a point that is thecause of the incident.

Concerning the watching system according to the first embodiment, a casein which a condition rule is set in advance, thereby setting a targetcondition corresponding to medical data has been described. However,since the relationship between a work and a target condition is notnecessary obvious, it is sometimes difficult to set a comprehensivecondition rule in advance. As a characteristic feature, the workmanagement system according to the fourth embodiment extracts a targetcondition corresponding to a work from work instruction data and adds acondition rule.

The work management system according to the fourth embodiment includesan arrangement common to the watching system according to the firstembodiment. Hence, in the fourth embodiment, portions different from thefirst embodiment will be described.

FIG. 13 is a block diagram showing an example of the arrangement of thework management system according to the fourth embodiment. As shown inFIG. 13, the work management system according to the fourth embodimentincludes a detection rule storage device 1, a terminal apparatus 10, asetting apparatus 20 c, a video supplying device 30, and a detectingapparatus 40. Note that these apparatuses may be connected via anetwork. As the network, for example, a fixed telephone line network, aportable telephone line network, the Internet, or the like isapplicable. In addition, these apparatuses may be included in one of theapparatuses.

The setting apparatus 20 c is an apparatus that sets a target conditionand a detection rule corresponding to work instruction information basedon work instruction data, and outputs them to the detection rule storagedevice 1, like the setting apparatus 20 according to the firstembodiment. The setting apparatus 20 c includes a condition rule storageunit 26 c and a target condition setting unit 24, and further includes acondition rule setting unit 21 c, a data input unit 22 c, a contentanalyzing unit 23 c, and a detection rule setting unit 25 c.

The condition rule setting unit 21 c sets a condition rule to output atarget condition from the work instruction information and outputs it tothe condition rule storage unit 26, like the condition rule setting unit21 according to the first embodiment. As a characteristic feature, ifthe work instruction information includes the information of the targetcondition, the condition rule setting unit 21 c automatically sets thecondition rule. For example, assume that work instruction data is anincident report as shown in FIG. 12, and work instruction informationincludes a work category “inject the powder into the sack” and a causeof incident “the lever for fixing the sack is not down”. In this case,the condition of the cause of the incident is defined as a targetcondition, thereby automatically setting the condition rule. Forexample, when the work category “inject the powder into the sack” isinput, a condition rule to output a target condition “the lever forfixing the sack is not down” can automatically be set.

The data input unit 22 c acquires work instruction data that is theinput to the setting apparatus 20 c, and outputs the work instructiondata to the content analyzing unit 23 c, like the data input unit 22according to the first embodiment. However, the data input unit 22 c canacquire work instruction data of a plurality of different forms.Furthermore, the data input unit 22 c estimates or acquires the categoryof the input work instruction data, and outputs it to the contentanalyzing unit 23 c. The categories of work instruction data are, forexample, work instruction data, an incident report, and the like. Thecategory of the work instruction data may be estimated by extracting acorresponding keyword in the work instruction data. Alternatively, thedata input unit 22 c may provide a GUI or physical keys used to inputthe category and cause the user of the system to input the category.

The content analyzing unit 23 c analyzes the contents of the workinstruction data, extracts work instruction information, and outputs itto the target condition setting unit 24 and the detection rule settingunit 25 c, like the content analyzing unit 23 according to the firstembodiment. Based on the category of the input work instruction data,the content analyzing unit 23 c changes the work information to beextracted. For example, if the work instruction data is a work standarddocument, the content analyzing unit 23 c extracts the category of thework, the work order, and notes during the work as work instructioninformation. In addition, if the work instruction data is an incidentreport, the content analyzing unit 23 c extracts the category of thework and the category of the cause of the incident as work instructioninformation.

The method of extracting work information by the content analyzing unit23 c is not limited to a specific method. For example, when workinstruction data of the form as shown in FIG. 11 is input, the contentanalyzing unit 23 c extracts a text representing each work such as“extract a sack from the box” from the work list in the work instructiondata, and converts it into a work category. As an example of theprocessing, a method of applying latent semantic analysis to a textrepresenting a candidate of a work category and an input text of a workand making selection based on the similarity of latent meanings can beused. Note that as the work category, a work category defined in advanceusing the same GUI as the GUI shown in FIG. 2, which is provided by thecondition rule setting unit 21 according to the first embodiment, isused. Next, the information of the work order is given to each convertedwork category. The information of the work category and the order isthus extracted as work instruction information. Note that the method ofextracting the work information is not limited to the example describedhere, and the work information may be extracted by another method.

The detection rule setting unit 25 c sets a detection rule based on thework instruction information input from the content analyzing unit 23 cand the target condition input from the target condition setting unit24, and outputs the detection rule to the detection rule storage device1. As a characteristic feature, the detection rule according to thisembodiment includes a work detection rule including a processing methodof detecting a work in a video and a parameter and a condition detectionrule to detect the target condition from the video of the work. At thetime of detection, the work is detected based on the work detectionrule, and the condition detection rule corresponding to the detectedwork is executed.

The processing method of the work detection rule is not limited to aspecific method. For example, the work may be detected by applying anaction recognition method or the work may be detected based on theinformation of a work schedule acquired from the work instructioninformation and time information. In addition, the condition detectionrule is set like the condition detection rule according to the firstembodiment.

With the above-described processing, the target condition during a workcan be detected from a video based on work instruction data. It is alsopossible to automatically set a condition rule based on the workinstruction data.

The embodiments described above can be applied to a monitoring system ina store. For example, assume that there exists a database that sharesthe information of shoplifting harm among a plurality of stores in theneighborhood. In this case, the monitoring system according to thisembodiment acquires the information of shoplifting harm from thedatabase, and sets a detection rule to detect a similar person based onthe outer appearance of a shoplifter and features of actions.Accordingly, it is expected that shoplifting attempted by the sameshoplifter in a neighboring store can be detected before it happens.

In addition, the present invention can also be implemented by executingthe following processing. That is, software (program) that implementsthe functions of the above-described embodiments is supplied to thesystem or apparatus via a network or various kinds of storage media, andthe computer (or CPU or MPU) of the system or apparatus reads out andexecutes the program.

As the industrial applicability of the present invention, the presentinvention can be used in a field to perform detection from a video suchas watching in a hospital or a care facility, a monitoring system, andincreasing the productivity in a factory.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-065498, filed Mar. 29, 2018 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus formonitoring an object in a video, comprising: a first storage unitconfigured to store a plurality of pieces of rule information defining atarget part and a moving amount of the target part to be detected in acondition; a second storage unit configured to store a plurality of IDseach specifying an object and the rule information corresponding to eachof the plurality of IDs; a third storage unit configured to store aplurality of IDs each specifying an object and feature data forrecognizing each of the objects; an input unit configured to inputinformation identifying a monitored object and information representinga condition of the monitored object; an acquiring unit configured toacquire, from the first storage unit, the rule information defining thetarget part and the moving amount of the target part using the conditioninputted by the input unit, as the rule information corresponding to themonitored object; a determination unit configured to receive, from acamera, a video captured by the camera, detect an object in the videoand determine whether the detected object in the video is the monitoredobject using the information identifying the monitored object inputtedby the input unit; and an output unit configured to, if thedetermination unit determines that the detected object in the video isthe monitored object, determine whether the detected object in the videomoves the target part more than the moving amount, which is representedby the rule information acquired by the acquiring unit, and output aresult of the determination, wherein the input unit inputs the conditionof the monitored object and an ID of the monitored object, wherein theacquiring unit registers the ID of the monitored object input by theinput unit and the rule information acquired from the first storage unitbased on the condition of the monitored object into the second storageunit, and wherein the determination unit acquires an ID of an object inthe video by recognizing the object in the video by referring to thethird storage unit, determines whether the second storage unit stores anID matching with the acquired ID, and determines that the object in thevideo is the monitored object if it is determined that the secondstorage unit stores the ID matching with the acquired ID.
 2. Theapparatus according to claim 1, wherein the monitored object is aperson, and the condition of the monitored object includes a healthcondition.
 3. The apparatus according to claim 2, wherein a movementindicated by the rule information includes a dangerous movement.
 4. Theapparatus according to claim 3, wherein the rule information includes athreshold of a danger level, and defines a movement which exceeds thethreshold as the dangerous movement.
 5. The apparatus according to claim2, wherein the input unit analyzes medical data and inputs the healthcondition.
 6. The apparatus according to claim 2, further comprising apatient database configured to store information used to identify apatient, wherein the output unit refers to the patient database based onfeature information obtained from an image of the person in the videoreceived from the camera, determines whether the person in the video isa patient, and upon determining that the person is a patient, startsmonitoring the person in the video.
 7. The apparatus according to claim1, wherein a health condition includes a disease name.
 8. The apparatusaccording to claim 1, wherein the monitored object is one of a personand a machine, and the condition of the monitored object is a stage of awork.
 9. The apparatus according to claim 8, wherein a movementindicated by the rule information includes an abnormal movement.
 10. Theapparatus according to claim 1, wherein the rule information includes anallowable range for a movement and defines a movement which exceeds theallowable range as the movement to be detected.
 11. The apparatusaccording to claim 1, wherein the rule information includes arestriction/prohibition for a movement and defines a movement whichviolates the restriction/prohibition as the movement to be detected. 12.The apparatus according to claim 1, further comprising a correction unitconfigured to correct the rule information.
 13. The apparatus accordingto claim 12, wherein the correction unit is further configured tocorrect the rule information after a predetermined period.
 14. Theapparatus according to claim 12, wherein the correction unit is furtherconfigured to correct the rule information based on an image capturingenvironment of the video.
 15. The apparatus according to claim 1,wherein the rule information is described in a text form.
 16. Theapparatus according to claim 1, further comprising a creating unitconfigured to set and create the rule information.
 17. The apparatusaccording to claim 1, wherein in a case in which the monitored objectmoves the target part more than the moving amount which is representedby the rule information acquired by the acquiring unit, the output unitoutputs, to a terminal set in advance, information representing that thecondition to be detected is obtained.
 18. A method of controlling aninformation processing apparatus that includes a first storage unitconfigured to store a plurality of pieces of rule information defining atarget part and a moving amount of the target part to be detected in acondition, a second storage unit configured to store a plurality of IDseach specifying an object and the rule information corresponding to eachof the plurality of IDs, and a third storage unit configured to store aplurality of IDs each specifying an object and feature data forrecognizing each of the objects, and that monitors an object in a video,the method comprising: inputting information identifying a monitoredobject and information representing a condition of the monitored object;acquiring, from the first storage unit, the rule information definingthe target part and the moving amount of the target part using thecondition inputted in the inputting, as rule information correspondingto the monitored object; receiving, from a camera, a video captured bythe camera, detecting an object in the video and determining whether thedetected object in the video is the monitored object using theinformation identifying the monitored object inputted in the inputting;and if it is determined that the detected object in the video is themonitored object, determining whether the detected object in the videomoves the target part more than the moving amount, which is representedby the acquired rule information and outputting a result of thedetermination, wherein in the inputting, the condition of the monitoredobject and an ID of the monitored object are inputted, wherein in theacquiring, the ID of the monitored object inputted in the inputting andthe rule information acquired from the first storage unit based on thecondition of the monitored object are registered into the second storageunit, and wherein in the determining, an ID of an object in the video byrecognizing the object in the video is acquired by referring to thethird storage unit, it is determined whether the second storage unitstores an ID matching with the acquired ID, and it is determined thatthe object in the video is the monitored object if it is determined thatthe second storage unit stores the ID matching with the acquired ID. 19.A non-transitory computer-readable storage medium storing a programwhich causes, when executed by a computer, the computer to execute stepsof a method of controlling an information processing apparatus thatincludes a first storage unit configured to store a plurality of piecesof rule information defining a target part and a moving amount of thetarget part of an object to be detected in a condition, a second storageunit configured to store a plurality of IDs each specifying an objectand the rule information corresponding to each of the plurality of IDs,and a third storage unit configured to store a plurality of IDs eachspecifying an object and feature data for recognizing each of theobjects, and that monitors an object in a video, the method comprising:inputting information identifying a monitored object and informationrepresenting a condition of the monitored object; acquiring, from thefirst storage unit, the rule information defining the target part andthe moving amount of the target part using the condition inputted in theinputting; receiving, from a camera, a video captured by the camera,detecting an object in the video, and determining whether the detectedobject in the video is the monitored object; and if it is determinedthat the detected object in the video is the monitored object,determining whether the detected object in the video moves the targetpart more than the moving amount, which is represented by the acquiredrule information and outputting a result of the determination, whereinin the inputting, the condition of the monitored object and an ID of themonitored object are inputted, wherein in the acquiring, the ID of themonitored object inputted in the inputting and the rule informationacquired from the first storage unit based on the condition of themonitored object are registered into the second storage unit, andwherein in the determining, an ID of an object in the video byrecognizing the object in the video is acquired by referring to thethird storage unit, it is determined whether the second storage unitstores an ID matching with the acquired ID, and it is determined thatthe object in the video is the monitored object if it is determined thatthe second storage unit stores the ID matching with the acquired ID. 20.A monitoring system for monitoring an object in a video, the monitoringsystem comprising: at least one image capturing device configured tocapture the object; and a monitoring device configured to monitor theobject, wherein the monitoring device comprises: a communication unitconfigured to receive a video captured by the at least one imagecapturing device; a first storage unit configured to store a pluralityof pieces of rule information defining a target part and a moving amountof the target part to be detected in a condition; a second storage unitconfigured to store a plurality of IDs each specifying an object and therule information corresponding to each of the plurality of IDs; a thirdstorage unit configured to store a plurality of IDs each specifying anobject and feature data for recognizing each of the objects; an inputunit configured to input information identifying a monitored object andinformation representing a condition of the monitored object; anacquiring unit configured to acquire, from the first storage unit, therule information defining the target part and the moving amount of thetarget part using the condition inputted by the input unit, as ruleinformation corresponding to the monitored object; a determination unitconfigured to receive, from the at least one image capturing device, thevideo captured by the at least one image capturing device, detect anobject in the video and determine whether the detected object in thevideo is the monitored object using the information identifying themonitored object; and an outputting unit configured to, if thedetermination unit determines that the detected object in the video isthe monitored object, determine whether the detected object in the videomoves the target part more than the moving amount, which is representedby the rule information acquired by the acquiring unit, and output aresult of the determination, wherein the input unit inputs the conditionof the monitored object and an ID of the monitored object, wherein theacquiring unit registers the ID of the monitored object input by theinput unit and the rule information acquired from the first storage unitbased on the condition of the monitored object into the second storageunit, and wherein the determination unit acquires an ID of an object inthe video by recognizing the object in the video by referring to thethird storage unit, determines whether the second storage unit stores anID matching with the acquired ID, and determines that the object in thevideo is the monitored object if it is determined that the secondstorage unit stores the ID matching with the acquired ID.